Goto

Collaborating Authors

 boolq hellaswag logiqa openbookqa winogrande


Recycling Pretrained Checkpoints: Orthogonal Growth of Mixture-of-Experts for Efficient Large Language Model Pre-Training

arXiv.org Artificial Intelligence

Numerous computational costs have been invested in existing well-trained checkpoints, but many of them remain underuti-lized due to engineering constraints or limited model capacity. To efficiently reuse this "sunk" cost, we propose to recycle pretrained checkpoints by expanding their parameter counts and continuing training. We propose orthogonal growth method well-suited for converged Mixture-of-Experts model: interpositional layer copying for depth growth and expert duplication with injected noise for width growth. To determine the optimal timing for such growth across checkpoints sequences, we perform comprehensive scaling experiments revealing that the final accuracy has a strong positive correlation with the amount of sunk cost, indicating that greater prior investment leads to better performance. We scale our approach to models with 70B parameters and over 1T training tokens, achieving 10.66% accuracy gain over training from scratch under the same additional compute budget. Our checkpoint recycling approach establishes a foundation for economically efficient large language model pretraining. The unprecedented success of large language models (LLMs) has been largely attributed to scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022), which suggest that increasing model size However, training these models from scratch demands enormous computational resources, and the exponential growth of this cost poses a fundamental barrier to further progress. Consequently, developing methods to scale models efficiently under constrained computational budgets has become a critical research challenge. Modern LLM development pipelines routinely produce smaller pre-trained model checkpoints and numerous intermediate artifacts from processes like hyperparameter tuning or preliminary evaluations. These models are often discarded once training concludes, leaving much of their potential unrealized due to inherent size constraints.